Skip to content

PDX-470/472: Add detail + fields params for field masking (Thread C)#167

Merged
mrdailey99 merged 2 commits into
developfrom
feature/pdx-470-472-field-masking
May 14, 2026
Merged

PDX-470/472: Add detail + fields params for field masking (Thread C)#167
mrdailey99 merged 2 commits into
developfrom
feature/pdx-470-472-field-masking

Conversation

@mrdailey99
Copy link
Copy Markdown
Collaborator

@mrdailey99 mrdailey99 commented May 13, 2026

Summary

  • PDX-472: Add detail (summary|standard|full) and fields (comma-separated dot-paths) params to provar_project_inspect, provar_connection_list, provar_qualityhub_display, and provar_qualityhub_testcase_retrieve
  • PDX-470: Implement shared fieldMask.ts utility (maskFields, parseFieldsParam) and detailLevel.ts stub (applyDetailLevel) used by the above tools
  • Add provar_testsuite_validate and provar_testplan_validate diff-mode with baseline_run_id param (via validationDiff.ts)
  • Add validationScore.ts shared utility
  • Fix TypeScript cast errors for DiffableViolation conversions (as unknown as DiffableViolation[])
  • Add cross-thread untracked file exclusions to .eslintignore, .mocharc.json, and test/tsconfig.json to prevent parallel-thread files from breaking CI on this branch

Motivation

Agents in iterative fix-validate loops were emitting full tool responses on every call, compounding token cost with no way to request a narrower view. The detail and fields params give agents a standard mechanism to request summary-only or sparse responses.

Test plan

  • yarn test:only — 1032+ passing, 1 pending, 0 failing
  • yarn compile — TypeScript clean (no errors)
  • node_modules/.bin/eslint src test — lint clean
  • Manual: call provar_connection_list with fields=connections.name,connections.type and verify only those sub-fields are returned
  • Manual: call provar_project_inspect with detail=summary and verify only summary fields are returned

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings May 13, 2026 14:52
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Quality Orchestrator

🟢 LOW · 18 / 100 · Touches: utils. All changed files have mapped tests.


🧪 Tests to Run · Running 4 of 49 tests

  • unit/mcp/connectionTools.test.ts
  • unit/mcp/projectInspect.test.ts
  • unit/mcp/qualityHubTools.test.ts
  • unit/mcp/fieldMask.test.ts
▶ Run command
npx vitest run \
  unit/mcp/connectionTools.test.ts \
  unit/mcp/projectInspect.test.ts \
  unit/mcp/qualityHubTools.test.ts \
  unit/mcp/fieldMask.test.ts

⚡ quality-orchestrator  ·  /qo stub <file>  ·  qo analyze-local

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends several MCP tools with response-shaping controls (detail and/or fields) to reduce token usage, and adds new shared utilities to support masking and validation scoring. It also introduces “diff-mode” plumbing for validation tools (via baseline_run_id/run_id) and updates test/configuration files accordingly.

Changes:

  • Add detail (summary|standard|full) and fields (comma-separated dot-paths) support to selected MCP tools, backed by new detailLevel + fieldMask utilities.
  • Add validation scoring helpers (validationScore.ts) and extend validation tools with detail plus diff/baseline concepts.
  • Update unit tests and test/lint configs to cover the new masking behavior and adjust ignored/excluded files.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
test/unit/mcp/qualityHubTools.test.ts Adds unit tests for detail/fields behavior in Quality Hub tools.
test/unit/mcp/projectInspect.test.ts New unit tests for detail/fields behavior in provar_project_inspect.
test/unit/mcp/fieldMask.test.ts New unit tests for maskFields and parseFieldsParam.
test/unit/mcp/connectionTools.test.ts Adds unit tests for fields masking in provar_connection_list.
test/tsconfig.json Reformats config and adds explicit excludes for certain test files.
src/mcp/utils/validationScore.ts Adds completeness scoring and next-action recommendation helpers.
src/mcp/utils/fieldMask.ts Introduces field masking + fields param parsing utility.
src/mcp/utils/detailLevel.ts Introduces a shared detail-level response shaping helper.
src/mcp/tools/testSuiteValidate.ts Adds detail, diff-mode parameters, run persistence, and scoring fields.
src/mcp/tools/testPlanValidate.ts Adds detail support and scoring fields to plan validation responses.
src/mcp/tools/testCaseValidate.ts Adds detail, diff-mode parameters, run persistence, and scoring fields.
src/mcp/tools/qualityHubTools.ts Adds detail + fields support to display/testcase-retrieve tool responses.
src/mcp/tools/projectInspect.ts Adds detail + fields support to project inspection output.
src/mcp/tools/connectionTools.ts Adds fields support to connection list output.
.mocharc.json Adds Mocha ignore entries for specific tests.
.eslintignore Adds ESLint ignore entries, including a src/ path.
Comments suppressed due to low confidence (5)

src/mcp/tools/testSuiteValidate.ts:110

  • This tool now adds new response-shaping behaviors (detail filtering + diff-mode via baseline_run_id, plus run_id/completeness fields), but there are no corresponding unit tests covering these code paths. Please add tests in test/unit/mcp/testSuiteValidate.test.ts to assert the summary output shape and the {added,resolved,unchanged_count,run_id} diff response when baseline_run_id is provided.
        detail: z
          .enum(['summary', 'standard', 'full'])
          .optional()
          .default('standard')
          .describe(
            'Response verbosity. "summary": name, scores, and stop signal only. "standard"/"full": full violations and per-test-case results (default).'
          ),
        baseline_run_id: z
          .string()
          .optional()
          .describe(
            'run_id from a previous call. When provided, returns only violations that are new or resolved since that run: { added, resolved, unchanged_count, run_id }. If not found, returns error BASELINE_NOT_FOUND.'
          ),
      },

src/mcp/tools/testCaseValidate.ts:103

  • detail and baseline_run_id introduce new output shapes (summary filtering and diff-mode response) plus persistence via run_id, but there are no unit tests asserting these behaviors. Please extend test/unit/mcp/testCaseValidate.test.ts to cover detail=summary (field set) and a baseline_run_id round-trip that yields {added,resolved,unchanged_count}.
        detail: z
          .enum(['summary', 'standard', 'full'])
          .optional()
          .default('standard')
          .describe(
            'Response verbosity. "summary": is_valid, scores, and stop signal only. "standard"/"full": full issues list (default).'
          ),
        baseline_run_id: z
          .string()
          .optional()
          .describe(
            'run_id from a previous call. When provided, returns only issues that are new or resolved since that run: { added, resolved, unchanged_count, run_id }. If not found, returns error BASELINE_NOT_FOUND.'
          ),
      },
    },

src/mcp/tools/testPlanValidate.ts:151

  • provar_testplan_validate now supports the detail parameter and adds completeness_score/recommended_next_action, but the existing unit tests don’t exercise the new detail=summary response shaping. Please add a test in test/unit/mcp/testPlanValidate.test.ts asserting that detail=summary retains only PLAN_VALIDATE_SUMMARY_FIELDS and that standard preserves the full payload.
        detail: z
          .enum(['summary', 'standard', 'full'])
          .optional()
          .default('standard')
          .describe(
            'Response verbosity. "summary": name, scores, and stop signal only. "standard"/"full": full violations and hierarchy results (default).'
          ),
      },
    },
    ({ plan_name, test_suites, test_cases, test_suite_count, metadata, quality_threshold, detail }) => {
      const requestId = makeRequestId();
      log('info', 'provar_testplan_validate', { requestId, plan_name });

      try {
        const threshold = quality_threshold ?? 80;
        const input: TestPlanInput = {
          name: plan_name,
          test_suites: test_suites ?? [],
          test_cases: test_cases ?? [],
          test_suite_count,
          metadata: metadata ?? {},
        };

        const result = validatePlan(input, threshold);
        const summary = buildHierarchySummary(result);

        const completeness_score = calcCompletenessScore(summary.test_cases_valid, summary.total_test_cases);
        const recommended_next_action = calcNextAction(completeness_score, false);

        const response = {
          requestId,
          completeness_score,
          recommended_next_action,
          ...result,
          summary,
        };

        const detailLevel = (detail ?? 'standard') as DetailLevel;
        const finalResponse = applyDetailLevel(response, detailLevel, PLAN_VALIDATE_SUMMARY_FIELDS);

        return {
          content: [{ type: 'text' as const, text: JSON.stringify(finalResponse) }],
          structuredContent: finalResponse,
        };

src/mcp/tools/testSuiteValidate.ts:69

  • This introduces persistent on-disk state under ~/.provardx/validation for every provar_testsuite_validate call. Writing into the user’s home directory can be problematic in restricted/CI environments and can pollute developer machines during unit tests. Consider making the storage location configurable (env var / config) and/or using os.tmpdir() for non-persistent runs while still enabling diff mode when persistence is available.
function suiteStorageDir(): string {
  return path.join(os.homedir(), '.provardx', 'validation');
}

src/mcp/tools/testCaseValidate.ts:70

  • This tool now writes validation run state under ~/.provardx/validation on every call. Persisting in the user’s home directory can fail in locked-down environments and can leave behind files during unit tests/CI. Consider making the storage directory configurable (env var / config) and documenting the on-disk behavior, or using os.tmpdir() when persistence isn’t required.
/** Storage dir for testcase diff runs (home-based, shared across projects). */
function tcStorageDir(): string {
  return path.join(os.homedir(), '.provardx', 'validation');
}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +17 to +24
import {
generateRunId,
saveRun,
hasAnyRun,
loadBaselineViolations,
computeDiff,
type DiffableViolation,
} from '../utils/validationDiff.js';
Comment on lines +30 to +37
import {
generateRunId,
saveRun,
hasAnyRun,
loadBaselineViolations,
computeDiff,
type DiffableViolation,
} from '../utils/validationDiff.js';
Comment on lines +19 to +26
export function maskFields(obj: unknown, fields: string[]): unknown {
if (Array.isArray(obj)) {
return obj.map((item) => maskFields(item, fields));
}

if (obj === null || typeof obj !== 'object') {
return obj;
}
Comment thread src/mcp/tools/projectInspect.ts Outdated
.optional()
.default('standard')
.describe(
'Response verbosity: "summary" returns only project_path, provar_home, and the summary object; ' +
Comment on lines 88 to +117
title: 'Validate Test Plan',
description:
'Validate a Provar test plan: checks for empty plans, duplicate suite names, oversized plans (>20 suites), plan completeness (objectives, scope, methodology, environments, acceptance criteria, test data strategy, risk assessment), and naming consistency. Recursively validates child suites and test cases. Returns quality score, plan-level violations, and full hierarchy results.',
'Validate a Provar test plan: checks for empty plans, duplicate suite names, oversized plans (>20 suites), plan completeness (objectives, scope, methodology, environments, acceptance criteria, test data strategy, risk assessment), and naming consistency. Recursively validates child suites and test cases. Returns quality score, plan-level violations, and full hierarchy results. Use completeness_score and recommended_next_action to determine whether to continue iterating.',
inputSchema: {
plan_name: z.string().describe('Name of the test plan'),
test_suites: z.array(suiteSchema).optional().describe('Test suites belonging to this plan'),
test_cases: z.array(testCaseSchema).optional().describe('Test cases directly in this plan (not in a suite)'),
test_suite_count: z
.number()
.int()
.min(0)
.optional()
.describe('Explicit suite count for size check (overrides counting test_suites)'),
metadata: metadataSchema,
quality_threshold: z
.number()
.min(0)
.max(100)
.optional()
.describe('Minimum quality score for a test case to be considered valid (default: 80)'),
detail: z
.enum(['summary', 'standard', 'full'])
.optional()
.default('standard')
.describe(
'Response verbosity. "summary": name, scores, and stop signal only. "standard"/"full": full violations and hierarchy results (default).'
),
},
},
({ plan_name, test_suites, test_cases, test_suite_count, metadata, quality_threshold }) => {
({ plan_name, test_suites, test_cases, test_suite_count, metadata, quality_threshold, detail }) => {
Comment thread .eslintignore Outdated
Comment on lines +2 to +5
**/*.d.ts
test/unit/mcp/startupTuning.test.ts
src/mcp/utils/tokenMeta.ts
test/unit/mcp/tokenMeta.test.ts No newline at end of file
Comment thread .mocharc.json Outdated
Comment on lines +7 to +8
"node-option": ["loader=ts-node/esm"],
"ignore": ["test/unit/mcp/startupTuning.test.ts", "test/unit/mcp/tokenMeta.test.ts"]
Comment thread test/tsconfig.json Outdated
]
} No newline at end of file
"include": ["./**/*.ts"],
"exclude": ["../node_modules", "./lib", "./unit/mcp/startupTuning.test.ts", "./unit/mcp/tokenMeta.test.ts"]
mrdailey99 added a commit that referenced this pull request May 13, 2026
… primitive guard, missing tests

RCA: CI failed because validationDiff.ts was imported by testCaseValidate and testSuiteValidate
but existed only as an untracked file, so the module was not found on CI. Additionally Copilot
flagged that maskFields leaked full primitive values when dot-paths were used, projectInspect
description omitted requestId from its summary field list, and detailLevel/validationScore
utilities lacked dedicated unit tests.

Fix: Commit src/mcp/utils/validationDiff.ts as a tracked file; guard maskFields against leaking
primitives on sub-path requests (omit the key instead); fix projectInspect summary description
to list requestId; add detailLevel.test.ts and validationScore.test.ts; add detail-param and
baseline_run_id tests to testSuiteValidate and testPlanValidate; restore cross-thread lint
exclusions in .eslintignore with an explanatory comment (temporary until sibling PRs merge).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@mrdailey99 mrdailey99 force-pushed the feature/pdx-470-472-field-masking branch from 2a7861d to ef98332 Compare May 13, 2026 16:24
mrdailey99 added a commit that referenced this pull request May 13, 2026
…ld masking tools

RCA: CLAUDE.md requires docs updates for every PR that adds or modifies tool parameters; PR #167 added fields/detail to 4 tools without updating docs/mcp.md
Fix: Added fields and detail input param rows for provar_project_inspect, provar_connection_list, provar_qualityhub_display, and provar_qualityhub_testcase_retrieve; updated output descriptions to note detail/fields behaviour

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mrdailey99 and others added 2 commits May 14, 2026 15:12
RCA: Iterative fix-validate loops emit full inventory and CLI raw output on every call,
compounding token cost with no standard way for agents to request a narrower response.

Fix: Add fields=<comma-dot-paths> and detail=summary|standard|full params to
provar_project_inspect, provar_connection_list, provar_qualityhub_display, and
provar_qualityhub_testcase_retrieve. Implement shared utilities fieldMask.ts and
detailLevel.ts. Standard default preserves backward compatibility.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ld masking tools

RCA: CLAUDE.md requires docs updates for every PR that adds or modifies tool parameters; PR #167 added fields/detail to 4 tools without updating docs/mcp.md
Fix: Added fields and detail input param rows for provar_project_inspect, provar_connection_list, provar_qualityhub_display, and provar_qualityhub_testcase_retrieve; updated output descriptions to note detail/fields behaviour

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@mrdailey99 mrdailey99 force-pushed the feature/pdx-470-472-field-masking branch from b979f7c to 8e850ca Compare May 14, 2026 20:17
@mrdailey99 mrdailey99 merged commit 1352f0f into develop May 14, 2026
4 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants